Skip to content

feat(otel-thread-ctx): Node.js OTEP-4947 thread-context writer - #9210

Merged
szegedi merged 6 commits into
masterfrom
otel-thread-context-writer
Aug 13, 2026
Merged

feat(otel-thread-ctx): Node.js OTEP-4947 thread-context writer#9210
szegedi merged 6 commits into
masterfrom
otel-thread-context-writer

Conversation

@szegedi

@szegedi szegedi commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a Node.js writer for the OpenTelemetry Thread Local Context Record (OTEP-4947), letting out-of-process readers (typically eBPF profilers) sample the active trace/span ID and a small attribute payload with no cooperation from the tracer at read time. Gated behind DD_TRACE_OTEL_CTX_ENABLED (default off).

The writer itself lives in @datadog/pprof (5.16.0+); this branch wires it into the tracer and publishes the accompanying OTEP-4719 process context via libdatadog-nodejs (0.12.1+).

Relevant PRs in other repos that this PR builds upon:

open-telemetry/opentelemetry-specification/pull/4947 OTel Thread Context Record specification
polarsignals/custom-labels/pull/16 ref implementation, has a good description of Node.js specifics
DataDog/pprof-nodejs/pull/347 pprof-nodejs implementation
DataDog/pprof-nodejs/pull/366 follow-up to above
DataDog/libdatadog/pull/2162 libdatadog improvements to OTel Process Context this PR needs
DataDog/libdatadog-nodejs/pull/135 libdatadog-nodejs bindings
DataDog/libdatadog-nodejs/pull/153 some more libdatadog-nodejs bindings

A note on future work: unifying CPU profiler context and OTel thread context

There's lots of similarities in span-related context management between the new writer in otel-thread-ctx.js and the in-process CPU profiler in profiler/wall.js. Two of the commits in this PR extract common functionality (storage-channels.js and web-tags-cache.js) for both, these are elaborated more on below. It would be possible to implement this so that wall.js no longer maintains its own context data, but always uses the OTel context instead. Java and PHP profilers already do this. For us, the biggest blocker is that OTel context record relies on Async Context Frame, and the CPU profiler still needs to support Node.js 22-23 where it's off by default, so we can't unify before our lowest supported version is 24 where ACF is on by default.

It also has some extra runtime cost, but that's only a minor aspect. wall.js currently establishes its context very cheaply by only retaining references to span-related objects, and deferring string conversion until profile serialization, so it only happens for those contexts (~6k of them/minute) that were captured with samples. In contrast, utf-encoded string data need to be written into the OTel thread context immediately for each created span, since we don't know when it will be captured by an external eBPF reader.

We will likely still do the unification, especially if both are enabled by default so the OTel context record is generated anyhow. We can either do the unification during dd-trace-js 6.x cycle but then wall.js will be more complex as it'll have to handle both kinds of contexts, or defer until 7.x next year when the minimum supported Node.js version will be 24 so we can drop wall.js own context and just use OTel.

What's in this branch (commit by commit):

Bump @datadog/libdatadog to 0.12.1

libdatadog 0.12.1 is a bug-fix bump over 0.12.0 which introduced some functionality we need.

Extract storage-channels module from wall profiler

Pull the dd-trace:storage:enter / :before / dd-trace:span:finish / :tags:update diagnostic-channel wiring out of the wall profiler into a shared packages/dd-trace/src/storage-channels.js, so both the wall profiler and the new thread-context writer can subscribe to the same normalized activation stream. No functional change to the wall profiler.

Extract shared web-tags cache from wall profiler

The OTel thread context writer will need to walk the started-spans chain per span to find the nearest web-server ancestor, just like wall profiler does. For thi reason, we extract the functionality into packages/dd-trace/src/web-tags-cache.js: a single Symbol on the span, one lazy walk per span, and a dd-trace:web-tags:resolved diagnostics channel that fires once per span at the moment a previously-empty answer transitions to a real value via dd-trace:span:tags:update.

Add OTEP-4947 thread context writer

After the first three preparatory commits, this is the actual new functionality.

  • packages/dd-trace/src/otel-thread-ctx.js: the writer. Subscribes to storage-channels; on each storage:enter, builds (or reuses) a ThreadContext from @datadog/pprof.otelThreadCtx for the active span, populates trace/span IDs plus a positional attribute array (index 0 = datadog.local_root_span_id, then datadog.trace_endpoint for web-server spans, datadog.thread_name, datadog.thread_id), and installs it via context.enter(). Handles span-drift (re-installs the same cached context when we switch spans and back) and span-finish (clears the writer if the record is still ours to avoid leaking stale state past enterWith-style activation). Late endpoint discovery is handled via the shared web-tags cache (see below): the writer subscribes to webTagsCache.resolvedCh and appends the endpoint attribute in place when the shared cache signals a transition.
  • packages/dd-trace/src/proxy.js: gated require('./otel-thread-ctx').start() after profiler init.
  • Config wiring: DD_TRACE_OTEL_CTX_ENABLED added to supported-configurations.json; the generated .d.ts picks it up.
  • scripts/docker/: a test:docker:otel-thread-ctx harness that builds inside node:24-bookworm (the writer is Linux+AsyncContextFrame-only; macOS dev machines fall through to the harness).
  • Test suite (packages/dd-trace/test/otel-thread-ctx.spec.js, 20 cases): start() gate matrix, on-enter build/skip/drift, span-finish clear-vs-leave, tags-update endpoint append, and the process-context helper.

Publish OTEP-4947 process-context metadata via process discovery

For OTel thread context record to work correctly, information also needs to be published in the process context.

Sets up the OTEP-4719 process context so an out-of-process reader can decode the on-the-wire records. Adds getThreadLocalMetadata() in otel-thread-ctx.js — pulls the snapshot from @datadog/pprof.otelThreadCtx.getProcessContextAttributes (schema-version string, attribute key map, V8 layout constants) and reshapes it into the napi ThreadLocalMetadata form. tracer_metadata.js passes it as the last positional arg to processDiscovery.TracerMetadata(...). Returns undefined if pprof is missing → publishes without a threadlocal block.

Platform / runtime scope

  • Linux + AsyncContextFrame only: the OTEP-4947 reader contract is ELF-TLSDESC, only meaningful on Linux; the writer requires V8's AsyncContextFrame (default in Node 24+; behind --experimental-async-context-frame on 22/23). On any other environment, start() logs and returns false — no runtime cost.
  • Optional dep: @datadog/pprof is an optionalDependency; if it isn't installed, the writer stays inert and no threadlocal block is published in the process context.

Test plan

  • yarn test:otel-thread-ctx — 20/20 pass
  • yarn mocha --timeout 60000 packages/dd-trace/test/tracer_metadata.spec.js — 11/11 pass
  • yarn mocha --timeout 60000 packages/dd-trace/test/profiling/profilers/wall.spec.js — 32/32 pass (unchanged from master after the shared-cache extraction)

Jira: PROF-15220

@dd-octo-sts

dd-octo-sts Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 8.04 MB
Deduped: 8.7 MB
No deduping: 8.7 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jul 3, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 1 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 98.57% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f66073e | Docs | Datadog PR Page | Give us feedback!

@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.57%. Comparing base (07a9eb4) to head (f66073e).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff            @@
##           master    #9210    +/-   ##
========================================
  Coverage   98.56%   98.57%            
========================================
  Files         969      972     +3     
  Lines      140152   140773   +621     
  Branches    12529    12050   -479     
========================================
+ Hits       138139   138760   +621     
  Misses       2013     2013            
Flag Coverage Δ
aiguard 57.47% <100.00%> (+<0.01%) ⬆️
aiguard-integration 55.68% <72.00%> (+<0.01%) ⬆️
apm-bucket-0 57.21% <100.00%> (+<0.01%) ⬆️
apm-bucket-1 63.33% <100.00%> (-0.01%) ⬇️
apm-bucket-2 62.18% <100.00%> (+<0.01%) ⬆️
apm-bucket-3 59.77% <100.00%> (+<0.01%) ⬆️
apm-capabilities-tracing 62.49% <89.73%> (+0.25%) ⬆️
apm-integrations-aerospike 56.25% <100.00%> (+<0.01%) ⬆️
apm-integrations-confluentinc-kafka-javascript 61.16% <100.00%> (+0.03%) ⬆️
apm-integrations-couchbase 56.69% <100.00%> (+<0.01%) ⬆️
apm-integrations-http 61.88% <100.00%> (+<0.01%) ⬆️
apm-integrations-kafkajs 61.69% <100.00%> (+<0.01%) ⬆️
apm-integrations-next 59.38% <100.00%> (+<0.01%) ⬆️
apm-integrations-prisma 58.49% <100.00%> (+<0.01%) ⬆️
appsec 72.07% <100.00%> (-0.04%) ⬇️
appsec-express_fastify_graphql 69.40% <100.00%> (-0.01%) ⬇️
appsec-integration 50.12% <72.00%> (+<0.01%) ⬆️
appsec-kafka_ldapjs_lodash 63.38% <100.00%> (-0.01%) ⬇️
appsec-mongodb-core_mongoose_mysql 66.84% <100.00%> (-0.01%) ⬇️
appsec-next 56.65% <100.00%> (+<0.01%) ⬆️
appsec-node-serialize_passport_postgres 66.26% <100.00%> (-0.01%) ⬇️
appsec-sourcing_stripe_template 64.69% <100.00%> (-0.01%) ⬇️
debugger 64.23% <100.00%> (+0.01%) ⬆️
instrumentations-bucket-0 51.70% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-1 59.63% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-10 60.87% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-11 61.52% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-12 51.61% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-13 52.45% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-14 51.72% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-2 52.93% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-3 53.58% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-4 58.70% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-5 49.42% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-6 60.25% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-7 51.90% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-8 58.38% <100.00%> (+<0.01%) ⬆️
instrumentations-bucket-9 57.23% <100.00%> (+<0.01%) ⬆️
instrumentations-instrumentation-couchbase 50.95% <100.00%> (+<0.01%) ⬆️
instrumentations-integration-esbuild 34.20% <0.00%> (-0.01%) ⬇️
llmobs-ai_anthropic_bedrock 62.84% <100.00%> (-0.01%) ⬇️
llmobs-bucket-1 61.32% <100.00%> (-0.01%) ⬇️
llmobs-openai 61.72% <100.00%> (+<0.01%) ⬆️
llmobs-openai-agents_vertex-ai 60.01% <100.00%> (+<0.01%) ⬆️
llmobs-sdk 66.75% <100.00%> (-0.01%) ⬇️
master-coverage 98.57% <100.00%> (?)
openfeature 55.66% <72.00%> (+<0.01%) ⬆️
openfeature-unit 53.22% <100.00%> (+<0.01%) ⬆️
platform-core_esbuild_instrumentations-misc 41.23% <100.00%> (+<0.01%) ⬆️
platform-integration 60.41% <72.00%> (+<0.01%) ⬆️
platform-shimmer_unit-guardrails_webpack 38.89% <100.00%> (+<0.01%) ⬆️
plugins-bucket-0 56.91% <100.00%> (+<0.01%) ⬆️
plugins-bucket-1 54.01% <72.00%> (+<0.01%) ⬆️
plugins-bucket-11 61.45% <100.00%> (+<0.01%) ⬆️
plugins-bucket-17 61.27% <100.00%> (+<0.01%) ⬆️
plugins-bucket-18 61.90% <100.00%> (+<0.01%) ⬆️
plugins-bucket-19 61.29% <100.00%> (+<0.01%) ⬆️
plugins-bucket-20 63.70% <100.00%> (-0.01%) ⬇️
plugins-bucket-4 58.29% <100.00%> (+<0.01%) ⬆️
plugins-bullmq_cassandra_cookie 61.35% <100.00%> (+<0.01%) ⬆️
plugins-cookie-parser_crypto_dd-trace-api 56.34% <100.00%> (+<0.01%) ⬆️
plugins-fetch_fs_generic-pool 58.20% <100.00%> (-0.04%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 64.12% <100.00%> (-0.01%) ⬇️
plugins-hapi_hono_ioredis 59.88% <100.00%> (+<0.01%) ⬆️
plugins-knex_langgraph_ldapjs 55.05% <100.00%> (+<0.01%) ⬆️
plugins-light-my-request_limitd-client_lodash 58.36% <100.00%> (+<0.01%) ⬆️
plugins-mariadb_memcached_mercurius 61.27% <100.00%> (+<0.01%) ⬆️
plugins-mongodb_mongodb-core_mongoose 59.24% <100.00%> (+<0.01%) ⬆️
plugins-multer_mysql_mysql2 58.83% <100.00%> (+<0.01%) ⬆️
plugins-nats_node-serialize_opensearch 60.37% <100.00%> (+<0.01%) ⬆️
plugins-passport-http_pino_postgres 58.60% <100.00%> (+0.03%) ⬆️
plugins-process_pug_redis 57.37% <100.00%> (+<0.01%) ⬆️
plugins-undici_url_valkey 57.97% <100.00%> (-0.04%) ⬇️
plugins-vm_winston_ws 59.57% <100.00%> (+<0.01%) ⬆️
profiling 61.66% <100.00%> (+0.17%) ⬆️
serverless-aws-sdk-aws-sdk 55.11% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-base-inject-field 50.93% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-bedrockruntime 54.64% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-client 56.21% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-dynamodb 55.48% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-eventbridge 49.72% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-kinesis 59.06% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-lambda 57.22% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-s3 55.57% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-serverless-peer-service 59.32% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-sns 59.86% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-sqs 60.28% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-stepfunctions 55.41% <100.00%> (+<0.01%) ⬆️
serverless-aws-sdk-util 51.46% <100.00%> (+<0.01%) ⬆️
serverless-bucket-0 54.06% <72.00%> (+<0.01%) ⬆️
serverless-bucket-1 58.84% <100.00%> (+<0.01%) ⬆️
test-optimization-cucumber 71.02% <100.00%> (+0.01%) ⬆️
test-optimization-cypress 64.81% <72.00%> (+0.10%) ⬆️
test-optimization-jest 72.38% <100.00%> (+<0.01%) ⬆️
test-optimization-mocha 72.09% <100.00%> (+0.10%) ⬆️
test-optimization-playwright-playwright-atr 59.85% <72.00%> (+0.02%) ⬆️
test-optimization-playwright-playwright-efd 59.98% <72.00%> (+0.02%) ⬆️
test-optimization-playwright-playwright-final-status 60.15% <72.00%> (+0.01%) ⬆️
test-optimization-playwright-playwright-impacted-tests 59.69% <72.00%> (+0.16%) ⬆️
test-optimization-playwright-playwright-reporting 60.84% <72.00%> (-0.09%) ⬇️
test-optimization-playwright-playwright-test-management 60.66% <72.00%> (-0.08%) ⬇️
test-optimization-playwright-playwright-test-span 59.90% <72.00%> (-0.05%) ⬇️
test-optimization-selenium 59.05% <72.00%> (-0.12%) ⬇️
test-optimization-testopt 57.59% <72.00%> (+0.08%) ⬆️
test-optimization-vitest 73.27% <100.00%> (+0.06%) ⬆️
test-optimization-vitest-browser 58.93% <72.00%> (+0.01%) ⬆️
test-optimization-webdriverio 65.42% <100.00%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pr-commenter

pr-commenter Bot commented Jul 3, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-12 12:51:19

Comparing candidate commit f66073e in PR branch otel-thread-context-writer with baseline commit 07a9eb4 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2309 metrics, 49 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-207.011ms; +222.328ms] or [-7.607%; +8.170%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-230.764ms; +221.459ms] or [-8.909%; +8.550%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-155.652ms; +163.678ms] or [-4.977%; +5.234%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-185.404ms; +181.091ms] or [-6.318%; +6.171%]

scenario:appsec-control-20

  • unstable execution_time [-121.851ms; +130.353ms] or [-7.269%; +7.776%]

scenario:appsec-control-24

  • unstable execution_time [-115.166ms; +119.022ms] or [-9.169%; +9.476%]

scenario:appsec-control-26

  • unstable execution_time [-124360.709µs; +123807.609µs] or [-9.952%; +9.908%]

scenario:appsec-iast-no-vulnerability-control-20

  • unstable cpu_usage_percentage [-4.850%; +5.325%]
  • unstable execution_time [-21567.897µs; +21649.630µs] or [-8.271%; +8.303%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

  • unstable execution_time [-8.027ms; +25.881ms] or [-3.104%; +10.007%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-20

  • unstable execution_time [-25.357ms; +30.841ms] or [-4.601%; +5.597%]

scenario:child_process-shell-string-24

  • unstable execution_time [-13.720ms; +18.614ms] or [-4.304%; +5.839%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-1856.025ms; +3106.757ms] or [-22.456%; +37.589%]
  • unstable execution_time [-1864.191ms; +3139.721ms] or [-20.765%; +34.972%]
  • unstable instructions [-15.6G instructions; +26.5G instructions] or [-23.220%; +39.306%]
  • unstable max_rss_usage [-6.451MB; +11.228MB] or [-4.067%; +7.079%]
  • unstable throughput [-882.476op/s; +516.357op/s] or [-24.020%; +14.054%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-2245.892ms; +751.262ms] or [-23.486%; +7.856%]
  • unstable execution_time [-2258.738ms; +778.755ms] or [-21.917%; +7.556%]
  • unstable instructions [-20.4G instructions; +6.8G instructions] or [-25.670%; +8.521%]
  • unstable throughput [-163.258op/s; +444.617op/s] or [-5.079%; +13.832%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-1642.570ms; +673.820ms] or [-19.957%; +8.187%]
  • unstable execution_time [-1645.043ms; +672.393ms] or [-18.435%; +7.535%]
  • unstable instructions [-13.9G instructions; +5.7G instructions] or [-20.711%; +8.545%]
  • unstable throughput [-215.089op/s; +437.888op/s] or [-5.821%; +11.850%]

scenario:debugger-line-probe-without-snapshot-24

  • unstable cpu_user_time [-1784.784ms; +3436.613ms] or [-21.006%; +40.446%]
  • unstable execution_time [-1770.936ms; +3421.995ms] or [-19.249%; +37.195%]
  • unstable instructions [-15.6G instructions; +29.3G instructions] or [-22.510%; +42.234%]
  • unstable max_rss_usage [-5.737MB; +12.624MB] or [-3.636%; +8.001%]
  • unstable throughput [-986.989op/s; +520.005op/s] or [-27.428%; +14.451%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-2606.286ms; +4150.265ms] or [-27.304%; +43.479%]
  • unstable execution_time [-2602.585ms; +4167.744ms] or [-25.341%; +40.581%]
  • unstable instructions [-23.5G instructions; +37.3G instructions] or [-29.573%; +46.897%]
  • unstable max_rss_usage [-8.741MB; +14.006MB] or [-5.506%; +8.822%]
  • unstable throughput [-820.234op/s; +510.588op/s] or [-25.414%; +15.820%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-283.236ms; +415.168ms] or [-5.834%; +8.551%]
  • unstable execution_time [-286.981ms; +417.092ms] or [-5.818%; +8.456%]
  • unstable throughput [-134501.250op/s; +108353.807op/s] or [-7.867%; +6.337%]

scenario:id-parse-128bit-20

  • unstable execution_time [-127.981ms; +194.086ms] or [-4.733%; +7.178%]

scenario:plugin-graphql-long-with-depth-and-collapse-off-20

  • unstable max_rss_usage [-16.019MB; +42.406MB] or [-4.024%; +10.652%]

scenario:plugin-graphql-long-with-depth-off-20

  • unstable max_rss_usage [-4.676MB; +12.589MB] or [-3.582%; +9.644%]

scenario:plugin-graphql-long-with-depth-off-26

  • unstable max_rss_usage [-34.441MB; +25.533MB] or [-18.656%; +13.831%]

scenario:plugin-graphql-long-with-depth-on-max-20

  • unstable cpu_user_time [-598.629ms; +572.409ms] or [-5.175%; +4.948%]
  • unstable execution_time [-619.395ms; +591.760ms] or [-5.247%; +5.013%]
  • unstable throughput [-3.460op/s; +3.622op/s] or [-5.076%; +5.314%]

scenario:plugin-pg-service-26

  • unstable cpu_usage_percentage [-10.030%; +7.805%]
  • unstable execution_time [-108.820ms; +143.467ms] or [-11.837%; +15.605%]
  • unstable throughput [-676678.749op/s; +545444.044op/s] or [-10.133%; +8.167%]

scenario:test-optimization-large-suite-20

  • unstable max_rss_usage [-4092.438KB; +3968.105KB] or [-5.196%; +5.039%]

@szegedi
szegedi force-pushed the otel-thread-context-writer branch 5 times, most recently from 797fe55 to d608888 Compare July 8, 2026 08:43
@szegedi
szegedi marked this pull request as ready for review July 8, 2026 09:23
@szegedi
szegedi requested review from a team as code owners July 8, 2026 09:23
@szegedi
szegedi requested review from BridgeAR and wconti27 and removed request for a team July 8, 2026 09:23

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d608888464

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

// the result on the span.
function getCachedWebTags (span) {
const cached = getCache(span)
if (cached.resolved) return cached.webTags

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Invalidate cached misses when parent web tags resolve

When a child span is entered before its HTTP parent receives route/resource tags, this cache stores resolved=true with webTags === undefined for the child. A later dd-trace:span:tags:update on the parent only publishes resolvedCh for the parent span, so re-entering the already-cached child hits this fast path and never re-walks to pick up the now-resolved endpoint; the OTel thread-context record (and wall-profiler context through the shared cache) will keep missing endpoints for common flows where routing tags are set after downstream spans start.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a valid issue, although it's been pre-existing for very long time (it existed before in wall.js code before we extracted it into web-tags-cache.js. This is rather non-trivial to fix, we need to either not-cache-negative-answers (extra walks each activation), or track a parent-children reverse map to invalidate on parent transition. Given the complexity, I find it acceptable as a corner case, but I can file a JIRA issue to keep track of it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread packages/dd-trace/src/otel-thread-ctx.js Outdated
// wins; subsequent calls are no-ops regardless of their argument. In
// practice all callers observe the same global ACF state.
function ensureChannelsActivated (asyncContextFrameEnabled) {
if (channelsActivated) return

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Allow later callers to request non-ACF hooks

If DD_TRACE_OTEL_CTX_ENABLED initializes these channels first on an ACF-capable runtime, this flag is set after only the enterWith wrapper is installed. A later wall-profiler start in auto mode with DD_PROFILING_ASYNC_CONTEXT_FRAME_ENABLED=false calls ensureChannelsActivated(false), but returns here before installing the async_hooks.before publisher and run() wrapper that non-ACF profiling relies on, so code-hotspot/endpoint contexts stop updating for that configuration.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true, but the scenario is quite unrealistic as it requires the combination of DD_TRACE_OTEL_CTX_ENABLED=1 DD_PROFILING_ASYNC_CONTEXT_FRAME_ENABLED=0 DD_PROFILING_ENABLED=auto. I'll file a JIRA as a low-priority follow-up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed for eventual follow-up as https://datadoghq.atlassian.net/browse/PROF-15354

Comment thread packages/dd-trace/src/tracer_metadata.js
szegedi added a commit that referenced this pull request Jul 8, 2026
…times

Addresses Codex review point #4 on #9210: getThreadLocalMetadata would
happily return a payload on macOS/Windows or on Node without an active
AsyncContextFrame, which would cause libdatadog to advertise a
threadlocal block that no writer is actually producing. Gate the
function on the same platform + ACF conditions start() already checks,
so callers see 'no threadlocal block'.
szegedi added a commit that referenced this pull request Jul 8, 2026
Addresses Codex review point #2 on #9210: an OTEP-4947 record can only
carry the first-written value for each attribute key, and HTTP plugins
routinely set 'http.method' up front and add 'http.route' (plus
'resource.name') later once routing has resolved. The writer previously
committed the endpoint on first activation, so a request to '/users/:id'
was recorded as 'GET' forever.

Introduce an isEndpointFinal(tags) heuristic and only write the endpoint
when the tag bag looks stable — either 'resource.name' is set, or both
'http.method' and 'http.route' are. Otherwise mark the context as
needsEndpoint and re-check on every 'dd-trace:span:tags:update' fire
(switched from webTagsCache.resolvedCh, which only fires on presence
transitions and would miss content-only updates).
@szegedi
szegedi force-pushed the otel-thread-context-writer branch from 50b56d7 to eb0ec07 Compare July 8, 2026 11:12
szegedi added a commit that referenced this pull request Jul 10, 2026
Addresses Codex review point #2 on #9210: HTTP plugins routinely set
'http.method' up front and add 'http.route' (plus 'resource.name')
later once routing has resolved. The writer previously committed the
endpoint on first activation, so an out-of-process reader sampling
mid-request saw a bare 'GET' as the endpoint for a call to
'/users/:id'.

OTEP-4947 duplicates are last-wins, so a later appendAttributes would
overwrite the interim value for readers that decode the record in
full — but a sampling reader can still observe the incomplete value.
Introduce an isEndpointFinal(tags) heuristic and only write the
endpoint when the tag bag looks stable — either 'resource.name' is
set, or both 'http.method' and 'http.route' are. Otherwise mark the
context as needsEndpoint and re-check on every
'dd-trace:span:tags:update' fire (switched from
webTagsCache.resolvedCh, which only fires on presence transitions and
would miss content-only updates).
@szegedi
szegedi force-pushed the otel-thread-context-writer branch 3 times, most recently from 1a19d66 to 275b832 Compare July 13, 2026 11:42
Comment on lines +3414 to +3420
"DD_TRACE_OTEL_CTX_ENABLED": [
{
"implementation": "A",
"type": "boolean",
"default": "false"
}
],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fingers crossed we can enable by default (on supported node.js versions) soon ;)

Comment on lines +60 to +66
// Positional attribute layout. The local root span ID stays at index 0 by
// convention (mirrors libdatadog's libdd-otel-thread-ctx, where
// `local_root_span_id` is always the first entry in
// `threadlocal.attribute_key_map`), encoded as a 16-character lowercase
// hex string. Endpoint, thread name, and thread id follow. Adding more
// means assigning the next index and updating ATTRIBUTE_KEYS
// accordingly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Since we're supplying a threadlocal.attribute_key_map I'm not sure if libdatadog will still prepend the local root span id, might be worth checking

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does prepend it. At least, the version we built libdatadog-nodejs process_discovery crate still did, if this changes in a later libdatadog release, we should update accordingly. I know there's some talk of retiring local root span id?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it worked when you did it, that's fine!

I know there's some talk of retiring local root span id

Yes, it's on my todo list to experiment a bit, but I'll share it loudly if it looks like we're going in that direction so yeah for now let's keep it.

@szegedi
szegedi force-pushed the otel-thread-context-writer branch from ba55aae to 1ebc38e Compare July 24, 2026 13:32
Comment thread package.json
Comment thread packages/dd-trace/src/otel-thread-ctx.js Outdated
Comment thread packages/dd-trace/src/otel-thread-ctx.js Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ef6f390d2c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/otel-thread-ctx.js
Comment thread packages/dd-trace/src/otel-thread-ctx.js Outdated
szegedi added a commit that referenced this pull request Aug 12, 2026
Detaching the finished span's ThreadContext from the current async-context
frame only covers that one frame. Sibling frames, and continuations the span
scheduled before finishing, inherit the same ThreadContext reference; in ACF
mode no `before` hook exists and no storage:enter fires in those frames to
overwrite the record, so an out-of-process reader kept seeing the finished
span as the active thread context there.

Use the new ThreadContext.invalidate() from @datadog/pprof 5.18.0, which
marks the record's `valid` byte 0 in place and so drops it out of scope for
every frame holding the reference at once. It runs unconditionally: the
frame calling span.finish() is not necessarily the frame holding that span's
context (a client span finished from a callback where the parent server span
is active, or any manual finish() from an unrelated context), and the record
dies with the span regardless of who holds it. clearContext() stays gated on
the current frame actually being the holder — that call is only about making
the record collectable rather than leaving an invalid one attached.

Reported by codex on #9210.
@szegedi
szegedi requested a review from IlyasShabi August 12, 2026 10:08

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 21754042d9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/otel-thread-ctx.js Outdated
Comment thread packages/dd-trace/src/otel-thread-ctx.js
szegedi added a commit that referenced this pull request Aug 12, 2026
The endpoint name a web-server span's tag bag yields changes as the request
progresses, and the two consumers of that bag disagreed about what to do
about it.

datadog-plugin-next starts its request span with `resource.name: req.method`
and only replaces it with `${req.method} ${page}` once the page is known. The
OTEP-4947 writer treated any `resource.name` as a settled value, so a Next
request's record committed a bare `GET` on first activation and ignored the
resolved route forever. The same hole existed for `http.route` arriving on a
span whose `resource.name` was still the bare method, because
endpointNameFromTags prefers `resource.name`: what got written was the
placeholder even though the route was known. Finality is now decided on the
computed value rather than on which tags are present, which covers both.

That predicate moves to profiling/webspan-utils.js next to
endpointNameFromTags, since the wall profiler needs it too: it resolves
endpoint labels lazily at serialization time and so never saw the problem,
but the fallback endpoint it snapshots mid-request for when the tag bag is
unreadable later is never refreshed once set, and could be pinned to a bare
`GET`.

Records built before the endpoint settles also have to be filled in
afterwards, and only the request span's own record was. A descendant resolves
to its nearest web-server ancestor's tag bag and gets its own record with its
own endpoint copy, but the tags update carrying the route is published for the
ancestor, which cannot enumerate its descendants — so a span entered during
the deferral window never got an endpoint at all. web-tags-cache now announces
the moment a request's endpoint settles on endpointResolvedCh, and the writer
keeps the records waiting on that announcement grouped by the tag bag they are
waiting on, filling in every one of them at once. Records whose span finished
in the meantime are skipped: onSpanFinished already invalidated them.

Driving that off the cache's transition channels rather than off
`dd-trace:span:tags:update` directly leaves one subscriber on that hot channel
instead of two, and removes the writer's ordering dependency on the cache
having processed an update before it reads the cache.

web-tags-cache had no spec of its own despite now having two consumers; this
adds one pinning both transitions and the activation refcount.

Reported by codex on #9210.
szegedi added a commit that referenced this pull request Aug 12, 2026
…prof

Both failures this prevents would have surfaced from inside a diagnostic-channel
subscriber, which runs inline with the tracer's hot path, so the exception would
have landed in application code.

The compatibility check gated the otelThreadCtx namespace but not the
ThreadContext methods the writer calls on it. An older or overridden
@datadog/pprof exposing the namespace without invalidate() would pass the gate
and then throw out of the span-finish path and up through DatadogSpan#finish()
the first time an activated span finished. The check now covers appendAttributes,
enter and invalidate as well, and names the missing member in the warning.

@datadog/pprof also decides whether AsyncContextFrame is available by inspecting
process.execArgv, and throws from enter() when it concludes it is not. That
disagrees with the feature detection behind isACFActive whenever the flag reached
Node by another route, which is reachable today:

  $ NODE_OPTIONS=--experimental-async-context-frame node -e '...'   # Node 22.23.2
  {"isACFActive":true,"execArgv":[],"pprofWouldThrow":true}

Node 22 and 23 accept the flag in NODE_OPTIONS (Node 24 rejects it, and does not
need it), and a worker thread created with an explicit execArgv loses it too. In
those processes the first span activation would have thrown from enter() into
whatever code triggered it. start() now installs and detaches one throwaway
context up front and declines to start if that fails, so an unusable pprof costs
a warning instead of an application-visible exception.

The execArgv inference is worth replacing with feature detection upstream in
pprof-nodejs; this check is what keeps a dd-trace release safe against any
version that still has it.

Reported by codex on #9210.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bea0d4f084

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/otel-thread-ctx.js
Comment on lines +124 to +128
const startedSpans = getStartedSpans(spanContext)
const rootContext = startedSpans.length ? startedSpans[0].context() : spanContext
// Only write the endpoint when its value has settled; otherwise leave a hole
// and wait for webTagsCache to announce that it has.
const webTags = webTagsCache.getCachedWebTags(span)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve trace ancestry after partial flush

For traces that hit partial flush before all descendants finish, span_processor.js prunes context._trace.started down to only still-active spans, so the original local root and any finished web-server ancestor can disappear while child spans keep running. In that state this path chooses the first remaining active span as datadog.local_root_span_id and webTagsCache.getCachedWebTags() can no longer walk back to the request span, causing OTEP records for the rest of a large/long-lived request to carry the wrong root id and lose the endpoint; the writer needs ancestry/root data that survives partial flush.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true, and the mechanism is exactly as described: _erase() in span_processor.js ends with trace.started = active, where active collects only spans whose _duration is still undefined. So when a flush happens while the local root has already finished, the root drops out, started[0] becomes the
oldest span still running, and a finished web-server ancestor stops being reachable for web-tags-cache's parent-chain walk.

It isn't specific to this writer, though. started[0]-is-the-local-root is a repo-wide convention shared by priority_sampler.js, span_format.js, span_context.js, event_plugins/event.js, the wall profiler's
local-root-span-id label, profiler.js's endpoint counting, and web-tags-cache.js's parent walk. (Yeah, profiler is a bit overrepresented.) Partial flush skews all of them the same way, in code that ships today; this writer inherits the behavior rather than introducing it.

That's also why I'd rather not fix it locally here. A writer-local workaround (memoizing the first root id seen per trace, say) would make OTEP records disagree with the wall profiler's labels about local_root_span_id for the same span, and two profilers reporting different roots is worse than both sharing one skew.

Two things bound the impact meanwhile: it needs a trace past DD_TRACE_PARTIAL_FLUSH_MIN_SPANS (default 1000) and a root that finished while descendants keep running; and web-tags-cache memoizes the resolved tag bag per span, holding the bag itself rather than re-walking, so any span that resolved its endpoint before the flush keeps it. Only spans first activated after a partial flush are affected.

The real fix is a local-root reference the tracer core maintains across flushes, applied to all of the consumers above at once, that's tracer-core work, and its own change. getStartedSpans in profiling/webspan-utils.js now documents the limitation at the one point every consumer reads, so the next person to touch it doesn't have to rediscover it.

Move the dd-trace storage diagnostics channels (storage:enter,
storage:before, span:finish, span:tags:update), the legacy-storage
enterWith/run shimmer, and the getActiveSpan helper out of
profiling/profilers/wall.js into a new packages/dd-trace/src/storage-channels.js.

No behavior change for the wall profiler. The extraction is to let a
forthcoming OTEP-4947 thread-context writer reuse the same channel
infrastructure without duplicating the shimmer or pulling in the
profiler module.

Adjust wall.spec.js's proxyquire stubs to replace the new module
instead of mocking datadog-core directly.
@szegedi
szegedi force-pushed the otel-thread-context-writer branch from bea0d4f to 63161b5 Compare August 12, 2026 12:25

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 63161b5c04

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/otel-thread-ctx.js
The wall profiler and the OTEP-4947 thread-context will both
walk each span's started-spans chain to find the nearest web-server
ancestor, each caching the answer under its own Symbol. Move the walk
and cache into packages/dd-trace/src/web-tags-cache.js.

- getCachedWebTags(span): lazy parent-chain walk, cached on a shared
Symbol.
- onSpanTagsUpdated(span): call from a tagsUpdate subscriber; if the
walk previously came up empty and the span is now a web-server span,
promote its tags into the cache. Returns true iff the cache
transitioned from undefined to a real value — a signal to consumers
that they should snapshot the new value into whatever they built
while the answer was undefined.

wall.js still snapshots webTags into its per-sample ProfilingContext
because label generation reads it from the sample-context ref (not from
the span); its tagsUpdate handler now refreshes that snapshot only when
the shared cache signals a transition.

No functional change. CODEOWNERS gets the new file scoped to
@DataDog/profiling-js.
New module packages/dd-trace/src/otel-thread-ctx.js that mirrors the
active trace ID, span ID and current endpoint into a thread-local
OTEP-4947 record. An out-of-process eBPF reader discovers the record
via the otel_thread_ctx_nodejs_v1 TLS symbol exported by the
@datadog/pprof addon.

Highlights:

- Gates on Linux + AsyncContextFrame (Node 24+ default, or Node 22/23
  with --experimental-async-context-frame). isACFActive from
  datadog-core/src/storage is the single source of truth for that
  check.
- One ThreadContext is allocated per span the first time it's
  activated and cached on the span via a Symbol slot. Re-entries in
  any async-context frame re-install the same wrap via setContext;
  `getContext() === cachedContext` is the JS-reference identity
  check that replaces any byte-level comparison (same allocation-
  churn fix as the wall profiler in dd-trace-js#8638).
- The record always carries the local root span ID (16-char hex,
  index 0 by libdatadog convention), thread name and thread id
  (stable per-thread, computed once at module load). The endpoint
  attribute is appended in place when a web-server ancestor is
  discovered via the span:tags:update channel.
- On span:finish, if the writer's record currently belongs to the
  finishing span, setContext(undefined) is called so an out-of-
  process reader doesn't keep seeing a finished span as the active
  thread context (matters in enterWith-style sticky activation).

Activation gate:

- DD_TRACE_OTEL_CTX_ENABLED (boolean, default false) is registered
  in supported-configurations.json as implementation A, exposed as
  config.DD_TRACE_OTEL_CTX_ENABLED. proxy.js calls
  otel-thread-ctx.start() iff the flag is set; the module itself
  starts in a no-op state otherwise.

Tests + docker rig:

- packages/dd-trace/test/otel-thread-ctx.spec.js covers start()
  gating, enter/skip behavior, span-drift round-trip, span-finish,
  and the late-tags append path with the wire-record attribute
  layout (15 cases).
- scripts/docker/{Dockerfile,run-otel-thread-ctx-spec.sh} build a
  node:24-bookworm image and run the spec against the locally built
  sibling pprof-nodejs; driven by `npm run test:docker:otel-thread-ctx`.

Forthcoming follow-up: publishing the corresponding
threadlocal.attribute_key_map via process discovery, which requires
bumping @DataDog/libdatadog.
Sets up the OTEP-4719 process context so an out-of-process reader can decode
the on-the-wire records the thread-context writer emits. The metadata is
published through libdatadog-nodejs's process-discovery napi crate: bumped
here to 0.12.0, which exposes the ThreadLocalMetadata substruct with the full
'threadlocal.*' block (attribute key map, schema-version string, and extra
KeyValues for reader-side layout constants).

The pieces:

- Add getThreadLocalMetadata() in otel-thread-ctx.js. Pulls the process-context
  snapshot from @datadog/pprof (its otelThreadCtx.getProcessContextAttributes
  is the source of truth for the schema-version string and V8 layout constants
  the reader needs) and reshapes it into the napi ThreadLocalMetadata form:
  { attributeKeys, schemaVersion, extraAttributes: [{ key, intValue|stringValue }] }.
  Returns undefined when @datadog/pprof isn't installed or doesn't expose the
  helper.

- Wire tracer_metadata.js to pass the substruct (or undefined) as the last
  positional arg to processDiscovery.TracerMetadata(...), replacing the flat
  threadlocalAttributeKeys array. Gated on the same DD_TRACE_OTEL_CTX_ENABLED
  flag that activates the writer.

- Bump the @DataDog/libdatadog optionalDependency from 0.10.0 to 0.12.0.
The spec is technically already picked up by the top-level 'packages/dd-trace/test/*.spec.js'
glob in test:trace:core:ci (apm-capabilities.yml), but the file is
profiling-team-owned per CODEOWNERS. Adding a dedicated step in
profiling.yml gives the owning team direct failure visibility in their
own workflow.
@szegedi
szegedi force-pushed the otel-thread-context-writer branch from 63161b5 to f66073e Compare August 12, 2026 12:39

@IlyasShabi IlyasShabi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@szegedi
szegedi merged commit 988a3a4 into master Aug 13, 2026
686 checks passed
@szegedi
szegedi deleted the otel-thread-context-writer branch August 13, 2026 10:14
szegedi added a commit that referenced this pull request Aug 13, 2026
The shared web-tags cache records "no web-server ancestor" for a span and
never revisits it, but that answer expires: plugins set `span.type` after
creating the span — TracingPlugin.startSpan activates it before addRequestTags
runs — so a child created in that window walks past an ancestor that is about
to become a web-server span, and caches a miss for a chain that is about to
have one.

Promotion can't find those descendants, since the walk only goes upwards. So a
promotion now bumps a generation counter, and an empty answer older than the
counter is walked again on the next lookup. Resolved answers are untouched (the
cached bag is the ancestor's live tag object), and a span with no parent is
stamped as permanently empty, since only its own promotion could change it and
onTagsUpdate already handles that. When a re-walk turns an empty answer into a
real one, the cache publishes resolvedCh for that span, the same announcement a
promoted span gets, so a consumer doesn't have to care which way the ancestry
appeared.

The counter lives on the trace, not in the module. A promotion can only
invalidate empty answers within its own trace, because the walk follows
`_parentId` through `_trace.started` and never leaves it — while a
process-global counter would have every HTTP request's promotion invalidate
empty answers in unrelated traces, making a long-lived non-web span re-walk its
chain once per request served elsewhere, from the storage-enter path.

Invalidation alone fixes nothing, because both consumers only ask the cache
while building their per-span state, and the spans this affects already have
theirs built. So each now asks again for state built from an empty answer:

- the OTEP-4947 writer re-checks on re-entry when a record was built with no
  web-server ancestor, and attaches the endpoint or enlists the record for the
  request's endpoint announcement. Guarded so a re-entrant announcement from
  inside the lookup can't append the endpoint twice.
- the wall profiler re-checks in #getProfilingContext when the snapshot it
  holds has no webTags, since #spanTagsUpdated only fires for a span promoted
  itself, never for descendants that walked past it beforehand.

Both re-checks cost two property reads plus, at most, the cache's own
generation compare — a walk only happens when a promotion in that trace has
actually invalidated something.

Two test-harness inaccuracies are fixed along the way, both of which had been
hiding behaviour rather than testing it: web-tags-cache.spec.js's makeSpan
returned a fresh object from context() per call, so spying on it counted calls
on a throwaway and the "walks the parent chain once" assertion held vacuously;
and wall.spec.js's makeChildSpan gave the child its own _trace object, so
parent and child were in different traces, which no real trace chunk is.

Reported by codex on #9210 and #9805.
dd-octo-sts Bot pushed a commit that referenced this pull request Aug 14, 2026
Mirrors the active trace ID, span ID, local root span ID and current endpoint
into a thread-local OTEP-4947 record, so an out-of-process eBPF reader can
attribute samples without going through the tracer. The record is discovered via
the otel_thread_ctx_nodejs_v1 TLS symbol exported by the @datadog/pprof addon,
and decoded using an OTEP-4719 process context published through libdatadog's
process discovery (threadlocal.* attribute key map, schema version, and the V8
layout constants a reader needs to walk into the record).

Off by default, behind DD_TRACE_OTEL_CTX_ENABLED. Requires Linux and an active
AsyncContextFrame (default from Node 24, opt-in on 22/23), and refuses to start
unless the installed @datadog/pprof exposes every API member the writer calls
and a context can actually be installed in this process — an unusable pprof
costs a log line rather than an exception thrown from a hot-path diagnostic
channel subscriber.

One ThreadContext is built per span on first activation and cached on the span,
so re-entry in another async-context frame re-installs the same reference rather
than allocating. On span finish the record is invalidated in place, which drops
it out of scope for every frame that inherited it — sibling frames and
continuations the span scheduled before finishing — since no later storage event
reaches those.

The endpoint is held back until its value settles: plugins publish interim
routing tags, and datadog-plugin-next seeds resource.name with the bare request
method, so publishing early would leave a reader attributing samples to "GET".
Once a request's endpoint resolves it is appended to every record built under
that request, the request span's own and each descendant's.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Aug 14, 2026
dd-octo-sts Bot pushed a commit that referenced this pull request Aug 14, 2026
Mirrors the active trace ID, span ID, local root span ID and current endpoint
into a thread-local OTEP-4947 record, so an out-of-process eBPF reader can
attribute samples without going through the tracer. The record is discovered via
the otel_thread_ctx_nodejs_v1 TLS symbol exported by the @datadog/pprof addon,
and decoded using an OTEP-4719 process context published through libdatadog's
process discovery (threadlocal.* attribute key map, schema version, and the V8
layout constants a reader needs to walk into the record).

Off by default, behind DD_TRACE_OTEL_CTX_ENABLED. Requires Linux and an active
AsyncContextFrame (default from Node 24, opt-in on 22/23), and refuses to start
unless the installed @datadog/pprof exposes every API member the writer calls
and a context can actually be installed in this process — an unusable pprof
costs a log line rather than an exception thrown from a hot-path diagnostic
channel subscriber.

One ThreadContext is built per span on first activation and cached on the span,
so re-entry in another async-context frame re-installs the same reference rather
than allocating. On span finish the record is invalidated in place, which drops
it out of scope for every frame that inherited it — sibling frames and
continuations the span scheduled before finishing — since no later storage event
reaches those.

The endpoint is held back until its value settles: plugins publish interim
routing tags, and datadog-plugin-next seeds resource.name with the bare request
method, so publishing early would leave a reader attributing samples to "GET".
Once a request's endpoint resolves it is appended to every record built under
that request, the request span's own and each descendant's.
@dd-octo-sts dd-octo-sts Bot mentioned this pull request Aug 14, 2026
pabloerhard pushed a commit that referenced this pull request Aug 17, 2026
Mirrors the active trace ID, span ID, local root span ID and current endpoint
into a thread-local OTEP-4947 record, so an out-of-process eBPF reader can
attribute samples without going through the tracer. The record is discovered via
the otel_thread_ctx_nodejs_v1 TLS symbol exported by the @datadog/pprof addon,
and decoded using an OTEP-4719 process context published through libdatadog's
process discovery (threadlocal.* attribute key map, schema version, and the V8
layout constants a reader needs to walk into the record).

Off by default, behind DD_TRACE_OTEL_CTX_ENABLED. Requires Linux and an active
AsyncContextFrame (default from Node 24, opt-in on 22/23), and refuses to start
unless the installed @datadog/pprof exposes every API member the writer calls
and a context can actually be installed in this process — an unusable pprof
costs a log line rather than an exception thrown from a hot-path diagnostic
channel subscriber.

One ThreadContext is built per span on first activation and cached on the span,
so re-entry in another async-context frame re-installs the same reference rather
than allocating. On span finish the record is invalidated in place, which drops
it out of scope for every frame that inherited it — sibling frames and
continuations the span scheduled before finishing — since no later storage event
reaches those.

The endpoint is held back until its value settles: plugins publish interim
routing tags, and datadog-plugin-next seeds resource.name with the bare request
method, so publishing early would leave a reader attributing samples to "GET".
Once a request's endpoint resolves it is appended to every record built under
that request, the request span's own and each descendant's.
pabloerhard pushed a commit that referenced this pull request Aug 17, 2026
Mirrors the active trace ID, span ID, local root span ID and current endpoint
into a thread-local OTEP-4947 record, so an out-of-process eBPF reader can
attribute samples without going through the tracer. The record is discovered via
the otel_thread_ctx_nodejs_v1 TLS symbol exported by the @datadog/pprof addon,
and decoded using an OTEP-4719 process context published through libdatadog's
process discovery (threadlocal.* attribute key map, schema version, and the V8
layout constants a reader needs to walk into the record).

Off by default, behind DD_TRACE_OTEL_CTX_ENABLED. Requires Linux and an active
AsyncContextFrame (default from Node 24, opt-in on 22/23), and refuses to start
unless the installed @datadog/pprof exposes every API member the writer calls
and a context can actually be installed in this process — an unusable pprof
costs a log line rather than an exception thrown from a hot-path diagnostic
channel subscriber.

One ThreadContext is built per span on first activation and cached on the span,
so re-entry in another async-context frame re-installs the same reference rather
than allocating. On span finish the record is invalidated in place, which drops
it out of scope for every frame that inherited it — sibling frames and
continuations the span scheduled before finishing — since no later storage event
reaches those.

The endpoint is held back until its value settles: plugins publish interim
routing tags, and datadog-plugin-next seeds resource.name with the bare request
method, so publishing early would leave a reader attributing samples to "GET".
Once a request's endpoint resolves it is appended to every record built under
that request, the request span's own and each descendant's.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants